Skip to content

Ease of use - #4

Merged
Ryu1845 merged 8 commits into
mainfrom
ease-of-use
Jul 15, 2026
Merged

Ryu1845 merged 8 commits into
mainfrom
ease-of-use

Conversation

@Ryu1845

@Ryu1845 Ryu1845 commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Improve packaging and usage for laymen users

Ryu1845 and others added 8 commits July 14, 2026 12:25
Add a retraining-free expert-pruning path for the MoE backbone, complementing
quantization as a size lever.

- zonos2-perplexity --prune-stats: export per-(layer,expert) hit count + MSAN
  (mean ‖down-input‖² over routed tokens, the (1,0,2) activation-norm score from
  arXiv:2606.15716), reusing the existing imatrix accumulators.
- prune-cli: gguf→gguf rewrite that gathers the kept experts (and matching
  router_mlp4/router_bias slices) into a smaller GGUF; pure per-expert memcpy so
  it composes with quantize-cli in either order.
- Runtime --prune-mask on zonos2-cli/zonos2-perplexity: apply a keep-set
  dynamically (additive -inf mask before softmax AND on the top-k scores), for
  sweeping the quality knee without rewriting GGUFs. Byte-identical to a baked
  prune-cli model.
- prune-policy.h: shared keep-set selector with four modes — --keep N (uniform),
  --mass-eps E (relative MSAN-mass threshold, non-uniform), --drop-lowest N
  (global lowest-MSAN), --drop-below-hits H; floored at each layer's top_k.
- build_moe reads the expert count per layer from ffn_up_exps->ne[2] (not
  hp.n_expert), so heterogeneous pruned models load and the softmax renormalizes
  over survivors.

Finding on this model: pruning degrades gracefully (~0.02 KLD/expert, no cliff)
but is dominated by quantization on quality-per-byte; non-uniform mass-targeting
is worse than uniform because top_k=1 makes per-layer routing diversity the
binding constraint. Tooling kept for future, less MoE-sensitive models.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add a runtime whole-block bypass for measuring depth (layer) pruning,
complementing the expert (width) pruning path. `zonos2_model.layer_skip`
marks transformer blocks to skip at graph-build time: build_moe/attention
are not built and the residual stream passes through unchanged (exact
identity), so both compute and the block's KV-cache are saved. Exposed via
`zonos2-perplexity --skip-layers L1,L2,...` for leave-one-out / span KLD
sweeps against a --kl-divergence base.

Phase-0 leave-one-out on out/zonos2-f16.gguf (28 blocks, vs ref-multi):
the profile is literature-consistent in shape (shallow blocks 0-2 and the
final block 27 catastrophic to drop, KLD 2.4-5.6; deep MoE cluster 21/23/
25/26 most redundant, KLD 0.23-0.28) but dominated by quant in magnitude —
the single most-droppable block (L25) costs KLD 0.229 (~q3k-cliff level)
for -3.6% blocks, vs q4k's 0.072 for -72% expert bytes. Generative KLD
collapses as the depth-pruning literature's MC-benchmark wins predict it
would ("Benchmark Illusion"), and there is no healing stage here. Knob
kept for future less-MoE-sensitive models; pruning stays parked.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
…GUF discovery)

Release archives now bundle a start-zonos2 launcher (.sh for Linux, .command
for macOS Finder, .bat/.ps1 for Windows) that downloads any missing GGUFs from
the now-public Zyphra/ZONOS2-GGUF repo via system curl (resumable, q6_k
default), launches zonos2-server with the right flags, and opens the web UI;
CI sed-stamps the backend default to gpu in the vulkan/metal archives and the
Windows artifact becomes a .zip for Explorer-native extraction.

Both binaries auto-detect dac.gguf / spk-encoder.gguf next to the backbone
(gated in the CLI so .npy outputs keep their behavior), accept --spk and
--spk-encoder as aliases, and print an actionable curl hint when a model file
is missing. New tests/launcher-smoke.sh covers download-skip, download from a
local http server, the optional-spk warning, and teardown on linux/macos CI;
the windows job parse-checks the ps1.

Co-Authored-By: Claude Fable 5 <[email protected]>
…bview)

Opt-in -DZONOS2_APP=ON target that runs the whole HTTP server in-process on an
ephemeral 127.0.0.1 port and points a native webview (WebView2 / WKWebView /
WebKitGTK, via saucer v8) at it, so the app UI is the exact web/tts_ui.html the
server already serves and the HTTP API stays reachable while the app runs. First
launch shows a setup page with native .gguf pickers; the choice is saved to the
platform config dir and later launches boot straight in. Server-style CLI args
pass through verbatim for power users.

server.cpp: main() split into server_run(...,ctl) + zonos2_server_main(); real
main() guarded by ZONOS2_APP_EMBED. --port 0 now binds an ephemeral port
(bind_to_any_port/listen_after_bind) and publishes it back through a
zonos2_server_ctl so the app can attach and stop the server. Transparent to
normal server/CLI use (readme-commands + launcher-smoke still green).

CI: new linux-app job (ubuntu-24.04, gcc-14, GTK4>=4.12); macos bumped to
macos-15 + Xcode>=16.3 and builds/signs Zonos2.app; windows bundles
zonos2-app.exe; release waits on linux-app. saucer's runtime SQLite state
(.saucer) is gitignored.

Toolchain floor is much higher than the rest of the project (CMake>=3.31,
C++23); everything else keeps the C++17/CMake 3.18 floor. Validated by building
zonos2-app in an ubuntu:24.04 container and driving it headlessly (xvfb+dbus):
ephemeral bind, /health, and /tts/generate through the embedded server all pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
saucer's WKWebView backend and saucer-embed's injected sources are Objective-C++.
saucer enables OBJC/OBJCXX inside its own FetchContent subdirectory, but that
scope doesn't set CMAKE_OBJCXX_COMPILE_OBJECT for the top-level directory where
zonos2-app is defined (embed injects .mm sources into it), so CMake fails at the
generate step with "CMAKE_OBJCXX_COMPILE_OBJECT not set" — macOS only (WebKitGTK
is plain C++, so Linux/Windows were unaffected). Enable OBJC/OBJCXX in the
top-level scope under -DZONOS2_APP=ON, guarded to APPLE.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The previous fix (683fb2d) enabled OBJC/OBJCXX inside the ZONOS2_APP block, but
that runs *after* add_subdirectory(ggml). ggml's Metal backend enables OBJC in
its own subdirectory scope first, so the later top-level enable_language(OBJC)
no-ops and CMAKE_OBJC_COMPILE_OBJECT is never set in the top-level scope where
zonos2-app is defined -> generate still failed, now on OBJC (OBJCXX was fine
because ggml never enables it, so our call was its first enabler).

Move the option() + the APPLE enable_language(OBJC/OBJCXX) above
add_subdirectory(ggml) so the root scope is the first enabler of both languages;
the compile rules then propagate down to ggml and to zonos2-app alike. Still
APPLE- and ZONOS2_APP-guarded, so Linux/Windows and non-app builds are untouched
(native reconfigure clean).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Voice cloning decodes the reference audio through ffmpeg; plain TTS never needs
it. Instead of requiring users to install ffmpeg, resolve it as $ZONOS2_FFMPEG
-> a binary next to the executable -> PATH (spk-encoder.cpp, via a new shared
exe-path.h), and have the start-zonos2 launcher and the desktop app download a
static ffmpeg on first run when none is present.

Sources (fetched at the user's runtime from the provider — nothing is
redistributed, and ffmpeg runs as a separate process): Linux/Windows use the
BtbN LGPL builds (ffmpeg-n7.1-latest-{linux64,win64}-lgpl, extracted from the
archive); macOS arm64 has no clean LGPL static build, so a pinned GPL binary
(eugeneware/ffmpeg-static b6.1.1 ffmpeg-darwin-arm64). Override with
ZONOS2_FFMPEG (a path) or ZONOS2_FFMPEG_URL (a single-binary source).

The launcher fetches into <model-dir>/bin and exports ZONOS2_FFMPEG; the app
fetches into its config dir on a detached thread so startup isn't blocked
(cloning fails gracefully until it lands). A system ffmpeg on PATH is used as-is
(ZONOS2_SKIP_PATH_FFMPEG=1 forces a download, for tests). Standalone zonos2-cli
does not auto-download — it resolves env/next-to-exe/PATH.

Validated: native resolution (env/PATH/bogus) + a real clone decode; launcher
ffmpeg-download smoke (single-binary + tar-extract paths); and an ubuntu:24.04
container build of zonos2-app with a headless auto-download into the config dir.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
exe-path.h pulls <windows.h>, and app.cpp includes it before <httplib.h>'s
<winsock2.h>. Left alone, <windows.h> drags in the legacy <winsock.h> (v1),
which hard-conflicts with <winsock2.h> — hundreds of sockaddr/fd_set
redefinition errors, Windows-only (server.cpp is immune because compat.h never
pulls windows.h, so httplib includes winsock first there).

Define _WINSOCKAPI_ (+ NOMINMAX, matching httplib) before <windows.h> in
exe-path.h. Use _WINSOCKAPI_ rather than WIN32_LEAN_AND_MEAN so the OLE/COM
headers saucer's WebView2 backend needs stay available. Also drop app.cpp's now
-dead <windows.h>/<mach-o/dyld.h> block (exe-path.h provides them). Linux app
build + ffmpeg smoke reconfirmed in the ubuntu:24.04 container.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@Ryu1845
Ryu1845 merged commit 0012a68 into main Jul 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant